Remote I/O Optimization and Evaluation for Tertiary Storage Systems through Storage Resource Broker

Large-scale parallel scientific applications are generating huge amounts of data that tertiary storage systems emerge as a popular place to hold them. SRB, a uniform interface to various storage systems including tertiary storage systems such as HPSS, UniTree etc., becomes an important and convenient way to access tertiary data across networks in a distributed environment. But SRB is not optimized for parallel data access: one SRB I/O call to storage systems must access a contiguous piece of data just like UNIX I/O. For many access patterns, this results in numerous small I/O calls which are very expensive. In this paper, we present a run-time library (SRB-OL) for optimizing tertiary storage access on top of SRB low level I/O functions. SRB-OL extends various state-of-the-art I/O optimizations that could be found in secondary storage systems to a remote data access environment via SRB. We also present a novel optimization scheme: superfile that can deal with large amounts of small files efficiently. We also incorporate a subfile technique and other features in SRB such as container, migrate, stage and purge into our SRB-OL. How to use these optimizations is decided by a Meta-data Management System (MDMS) [7] that resides one level above SRB-OL. The user provides access pattern information/hints through user application to MDMS, and then MDMS uses these hints to choose an optimal I/O approach and passes the decision to SRB-OL. Finally, SRB-OL performs optimized SRB I/O calls to access data residing on tertiary storage systems. To give a quantitative view of optimized SRB I/O functions, we propose a performance model based on significant I/O experiments. By using this performance model, we can prove that collective I/O, superfile etc have significant performance improvements. In addition, we present an I/O Performance Predictor that can estimate I/O cost before the user actually carries out her experiment. This provides the user a lot of benefits for running her application.